Medical mapping system

ABSTRACT

Apparatus and method for sensing at an anatomic body site and mapping or transforming the sensor signal into various forms of virtual image and feedback signals, having particular application in assisting surgeons and other operators during a medical procedure. In one embodiment, a medical system is provided that includes a medical implement, a manipulator controllable by an operator for control of the medical implement at an anatomic body site and a sensing device for sensing a non-visible field associated with a body structure at the site. A controller, intercoupling the sensing device and the manipulator, includes a mapping component for translating characteristics of the sensed field signal into a tactile feedback signal to the manipulator to warn the surgeon that he is approaching this structure with the implement. Alternatively or in addition, a virtual image of the body structure is displayed, separate or preferably together with a visual image of the site, to assist a surgeon in manipulating a medical implement.

RELATED APPLICATIONS

[0001] This application is a continuation-in-part of and claims thebenefit of priority of:

[0002] U.S. Provisional Application Serial No. 60/257,816, filed Dec.21, 2000;

[0003] U.S. application Ser. No. 10/011,449, filed Nov. 16, 2001; and

[0004] U.S. application Ser. No. 10/011,450, filed Nov. 16,2001

[0005] each of which are hereby incorporated by reference in theirentirety.

FIELD OF THE INVENTION

[0006] The present invention relates in general to a mapping system andmethod useful for mapping identifying anatomic members such as tissues,nerves, vessels or any other anatomic body part. This mapping can assistin the manipulation of a medical instrument, catheter, or the like,either under direct or robotic control.

BACKGROUND OF THE INVENTION

[0007] Part of the “art” of surgery is knowing where to cut and wherethe “danger zones” are located. The surgeon relies on visual clues todetermine where important anatomical structures are located. He alsorelies upon palpation. Landmarks such as bony prominence, ligamentinsertions, position of muscles and solid organs, and other landmarksguide the surgeon during tissue dissection (also referred to astransection).

[0008] One of the major complications of surgery is inadvertenttransection of sensory nerves, motor nerves, arteries, veins and hollowviscus. In most cases this occurs when key structures are poorlyvisualized by the surgeon, or due to inexperience or altered anatomicalstructures (e.g., surgical landmarks). Blunt dissection is performedeither manually using the fingers or using an instrument, and sharpdissection is performed using a scissors, electrical-surgical device, orscalpel.

[0009] A common side effect or complication of much surgery isneurological trauma, paresis or permanent paralysis occasioned by thesevering of nerves. Over 5% of major head and neck surgeries (e.g.,parotidectomy) result in damage to the facial nerve resulting inweakness of the facial muscles or partial paralysis of the face. Afrequent consequence of prostate surgery is damage to the sacral nerves,which control erectile function and ejaculation. Almost 40% of menundergoing prostatectomy are left impotent or have significant problemswith erectile function.

[0010] Inadvertent perforation of the aorta, or of the superior orinferior Vena Cava, can cause a major hemorrhage which may result indeath. The Vena Cava may be ruptured during dissection of the spleen orresection of tissues on the posterior surface of the liver. Transectionof smaller arteries and veins, while less life-threatening, are a causeof significant morbidity, often require transfusion to compensate forblood loss, and can significantly increase the length of hospitalizationfollowing a procedure. Similarly, transection of the ureter results inmajor morbidity, requires multiple operations to correct, may result inrenal failure and significantly impairs quality of life. Perforation ofother hollow viscus, such as the bowel or bladder wall, may result inperitonitis and death.

[0011] Electrical transducers are commonly used in medical practice tomeasure electrical impulses within nerve bundles. These instrumentsinclude: electroencephalograms (EEG), electrocardiogram (EKG),electromyography (EMG), and others. All these devices have in common theability to measure electrical impulses generated by nerve structures.The electrical fields vary with motor nerve, sensory nerve, the degreeof myelinization and the number of nerves in the particular neuralbundle. The electrical impulses are typically in the form of a waveformthat is examined and interpreted by a surgeon or physician. Such areadout is primarily used as a diagnostic tool in determiningirregularities in organs such as the heart or brain.

[0012] Subtle temperature variations also exist within differentanatomical compartments. Venous blood returning from an extremity iscooler than arterial blood. Sites of infection tend to be slightlywarmer than healthy tissues. Abscess pockets tend to be somewhat coolerthan surrounding tissues. These temperature variations are generallybelow the threshold of human proprioception. The variations can be lessthan a tenth of a degree. A number of different thermistors, which canmeasure temperature variation to a hundredth of a degree, are used inmedicine to monitor vital signs, cardiac output, and other functions.Again, such use is primarily diagnostic.

[0013] Gallium and technetium, as well as other isotopes, are used fortracking infection, lymphatic drainage and to provide contrast duringmagnetic resonance imaging (MRI). These isotopes are gamma emitters and,in general patients undergoing these scans are evaluated by gammacounters which then measure the degree of radioactive uptake. This datais then graphically displayed as an x-ray film.

[0014] During sentinel node biopsy, lymphoscintigraphy is used tolocalize the regional draining nodal basin in breast cancer and melanomatherapy. A small incision is then made in the skin overlying the “hot”node. Blunt and sharp dissection is used to reach the nodal basin. Asterile-wrapped Geiger counter is then inserted into the wound toidentify the affected node(s). Further dissection and Geiger countertesting is performed until the radioactive sites are fully identifiedand resected. This is a time-consuming and laborious process, with theoperative field landmarks and lymph nodes shifting in space with eachinsertion/removal of instruments.

[0015] In these various techniques, there is a need for a better methodof identifying anatomic structures or parts, particularly when thesestructures or parts are not visible to the surgeon's eye. By identifyingthese structures or parts, one can avoid the aforementioned problemsthat can lead to tissue, organ, or nerve damage, paresis, permanentparalysis, or other injuries that impair the quality of life of thepatient.

SUMMARY OF THE INVENTION

[0016] In one embodiment, what is sensed is a non-visible field and whatis provided is a combined display of virtual and visual images. In thisembodiment, a system is provided for generating the display of a bodystructure. The system includes a sensor positionable at an internal bodysite for sensing a non-visible field of a body structure at the site andgenerating a sensor signal indicative of the field. A transformationsystem transforms the sensor signal into virtual image data. A source ofvisual image data for the site is also provided. A visual system enablescombined display of the visual image data and the virtual image data.

[0017] According to a corresponding method, steps are provided forsensing at an internal body site a non-visible field of a body structureat the site and generating a sensed signal indicative of the field,transforming the sensed signal into virtual image data, providing visualimage data for the site, and displaying in combination the visual imagedata and the virtual image data.

[0018] In another embodiment, what is sensed is a non-visible field by asensor positioned with a computer-controlled instrument, and what isprovided is virtual image data. In this embodiment, a system is providedfor obtaining virtual image data of a body structure. The systemincludes a computer-controlled instrument for positioning a sensor at aninternal body site, the sensor sensing a non-visible field of a bodystructure at the site and generating a sensor signal indicative of thesite, and a transformation system is provided for transforming thesensor signal into virtual image data.

[0019] According to a corresponding method, steps are provided forpositioning by computer control a sensor at an internal body site,sensing a non-visible field of a body structure at the site andgenerating a sensor signal indicative of the field, and transforming thesensor signal into virtual image data.

[0020] In another embodiment, what is sensed is a non-visible field andwhat is provided is a tactile feedback for controlling manipulation of amedical implement. In accordance with this embodiment, a system isprovided for controlling manipulation of a medical implement including asensor positionable at an internal body site for sensing a non-visiblefield of a body structure at the site and generating a sensor signalindicative of the field, a transformation system for transforming thesensor signal into a feedback signal, and a control system, including ahaptic user interface, for manipulating a medical implement at the site,the control system receiving the feedback signal and in response theretoproviding a tactile signal at the user interface.

[0021] In accordance with the corresponding method, there are stepsprovided for sensing a non-visible field of a body structure at aninternal body site and generating a sensed signal indicative of thefield, transforming the sensed signal into a feedback signal, andutilizing the feedback signal to provide a tactile signal at a hapticuser interface for controlling manipulation of a medical implement atthe site.

[0022] In another embodiment, what is sensed is a visual image and whatis provided is a tactile feedback for controlling manipulation of amedical implement. In accordance with this embodiment, a system isprovided for controlling manipulation of a medical implement including asensor positionable at an internal body site for sensing a visual imageof a body structure at the site and generating a sensor signalindicative of the image, a transformation system for transforming thesensor signal into a feedback signal, and a control system, including ahaptic user interface, for manipulating a medical implement at the site,the control system receiving the feedback signal and in response theretoproviding a tactile signal at the user interface.

[0023] In accordance with the corresponding method, there are stepsprovided for sensing a visual image of a body structure at an internalbody site and generating a sensed signal indicative of the image,transforming the sensed signal into a feedback signal, and utilizing thefeedback signal to provide a tactile signal at a haptic user interfacefor controlling manipulation of a medical implement at the site.

[0024] Additional Embodiments

[0025] Various aspects of the present invention are described herein ina number of different embodiments. Many of these embodiments relate to amapping system and method for identifying the location of certainanatomic areas such as, but not limited to, nerves or nerve bundles.This mapping involves the translation of certain sensed parameters (alsoreferred to as stimuli) into response parameters. The sensed parametersinclude non-visible data (electromagnetic, thermal, infra-red,radioactive, etc.), and in other embodiments visible data and/or forcesensed data. The response parameters include a display device, anaudible device, a vibratory device and force feedback to a hapticinterface.

[0026] Identification of the anatomic area may be used for the purposeof avoiding the area (such as in guiding a tissue dissection), or forthe purpose of targeting, locating or guiding toward the anatomic area.Also, the identification may be employed in either a robotic system or anon-robotic system (e.g., in association with a hand-held instrument,such as a laparoscopic instrument). A telerobotic system usually iscomprised of master and slave stations including, on the master side, ahaptic interface and a display device, and, of the slave side, acontrollable medical implement. Here a sensor is provided on the slaveside, at the anatomic site of interest. The sensor detects, inter alia,electric, thermal, infra-red and radioactive fields, and is preferably,but not necessarily, carried at the distal end of the medical implement.The sensing may be by manual scanning or may be under computer control.Furthermore, the sensing may be either to establish a database,particularly for mapping non-visible fields to a display, or may be usedin a real time mode, particularly for feedback control to the masterhaptic interface.

[0027] One aspect of the present invention is to provide, in addition toa visual image of an anatomic site of interest, a virtual image orboundary of an invisible (also referred to as non-visible) anatomicstructure at the site, such as nerves or nerve bundles. The virtualimage or boundary may be established by sensing, for example, anelectric field, thermal field, or radioactive field associated with theanatomic structure. The sensed field may also be used in “real time” tocontrol, for example, a haptic (relating to or based on the sense oftouch) interface by providing tactile feedback to the surgeon regardingthe hidden anatomic structure.

[0028] In accordance with one aspect of the invention there is provideda system for sensing a non-visible field to provide a tactile feedbackto a haptic interface. This system may include a surgical implement,including for example an end effector or tool, to perform apredetermined surgical procedure at an anatomic body site. A controlleris disposed between the surgical implement and the haptic interface anda sensor preferably, but not necessarily carried by the end effector,for sensing a non-visible field at the body site. The controller isresponsive to manipulations at the haptic interface by an operator tocontrol the surgical implement in performing the surgical procedure. Thecontroller also is responsive to a feedback signal which is a functionof a sensed parameter of the non-visible field. This signal is fed backto the haptic interface to provide a tactile indication to the operatoras to the location of the anatomic structure that generated the field.

[0029] In accordance with various aspects of the present invention, anon-visible field may comprise an electrical field, a thermal field, aradio-active field, or an IR field. Also, considered as falling withinthe scope of the present invention are any other non-visible fields.Generally speaking, by the term “non-visible” reference is being made toa particular field established in the anatomy, either inherent or viastimulation, and that is not observable by the surgeon by the naked eye,such as by way of an endoscope. A “field” refers to a series of datapoints sensed over a particular area and representative of suchparameters as electric field, temperature, or radio-activity. A “field”may also be identified as directly sensed or inferred sensoryinformation which constructs a data set of values.

[0030] “Non-contact” as used in connection with haptic feedback controlrefers to non-visible as well as other types of sensed effects that arenot based upon direct physical contact between the sensor and theanatomic member or part.

[0031] In accordance with various aspects of the invention, the sensedor predetermined parameter may comprise a magnitude of the field. Also,the sensor may be a voltmeter, thermistor, gamma detector, IR detector,or any other type of sensing device that would sense the particularnon-visible field. The aforementioned anatomic body member may comprisea nerve or nerve bundle, vascular vessels, or virtually any other bodypart.

[0032] In accordance with still another aspect of the invention, theremay be provided a stimulator for stimulating the anatomic body member.The stimulator may be controlled from the controller so as to emit apredefined electrical stimulation signal that is coupled to energize thebody member. The detected stimulation signal is fed back to thecontroller to identify the body member based upon a recognizablepattern. The pattern may be a repetitive pattern establishing ananatomic body member signature.

[0033] In accordance with another aspect of the invention, the systemmay include a display, which may be coupled with an optic element, fordisplay the area at the anatomic body site. A controller receives asignal from the sensor and establishes on the display a map of the field(a virtual boundary defined by the map) about an anatomic body member.Moving the sensor to a plurality of locations in juxtaposition to theanatomic body member generates a map including boundaries of the fieldrelative to the anatomic body site. The display is observable by thesurgeon for displaying both a visual image and a virtual image, thevirtual image being representative of the map boundaries and essentiallyrepresentative of the position of the non-visible field.

[0034] In accordance with another aspect of the invention, there isprovided a medical system comprising a surgical implement including anend effector used to carry out a predetermined surgical procedure at ananatomic body site, a sensor for sensing a non-visible field that isestablished associated with a member at the anatomic body site, and adisplay. Further included is a controller for receiving signals from thesensor and establishing a map of the field about the member at theanatomic body site. The controller couples to the display andestablishes on the display both a visual image of the area at theanatomic body site as well as a virtual image representative of aboundary defined by the map and thus defined by the non-visible field.

[0035] In accordance with another aspect of the invention, there mayalso be provided a haptic interface for control of the surgicalimplement. A controller coupled to the haptic interface controls anoperator's action at the haptic interface as a function of apredetermined parameter of the non-visible field.

[0036] In accordance with another aspect of the invention, there isprovided a telerobotic surgery system having a slave station at which asurgical implement is disposed and a master station with a surgeoninterface at which manipulations occur to control the surgicalimplement. Apparatus is provided for sensing a non-visible fieldassociated with an anatomic body member in which the non-visible fielddistinguishes the anatomic body member from the surrounding area. Theapparatus can be used for generating a tactile feedback signal at thesurgeon interface. Such apparatus may comprise a sensor carried by thesurgical implement for detecting the non-visible field generated fromthe anatomic body member. Also included is a controller coupled from thesensor, responsive to the magnitude of the field, for feeding back tothe master station a control signal to the surgeon interface so as toprovide a tactile response to the surgeon based upon the magnitude ofthe detected field.

[0037] In accordance with another aspect of the invention, there isprovided a telerobotic surgery system having a slave station at which asurgical implement is disposed and a master station with a surgeoninterface at which manipulations occur to control the surgicalimplement. An apparatus is provided for mapping a non-visible fieldassociated with an anatomic body part and in which the non-visible fielddistinguishes the anatomic body part from other anatomic structures.This apparatus includes a sensor for detecting the non-visible fieldgenerated from the anatomic body part, a controller coupled from thesensor, responsive to the disposition of the sensor at a plurality oflocations in juxtaposition to the anatomic body part, to generate a mapincluding boundaries of the field relative to other anatomic bodystructures, and a display (coupled to the controller) observable by thesurgeon for displaying both a visual image of the area about theanatomic body part as well as a virtual image representative of the mapboundaries.

[0038] In accordance with another aspect of the invention, there isprovided a medical device that comprises a surgical implement having anend effector used to carry out a predetermined surgical procedure at ananatomic body site that includes an anatomic body member that is notvisible to the surgeon. The medical device includes a handle memberoperated so as to control the surgical implement and end effector, asensor for detecting a non-visible field that is associated with theanatomic body member and that demarcates a boundary of the field of theanatomic body member, and a controller coupled from and responsive tothe sensor for providing at the handle a tactile indication to thesurgeon based on a predetermined parameter of the field.

[0039] In accordance with another aspect of the invention, there isprovided a medical system useful in identifying an internal anatomicbody member that is not visible to the surgeon by direct or indirectoptical means. This system enables improved control by the surgeon ofcertain surgical procedures. The system comprises a sensor for sensing anon-visible field that is associated with an anatomic body member. Byway of example, this non-visible field may be an electrical field abouta nerve or nerve bundle. A controller is employed coupled from thesensor and responsive to signals representative of different locationsof sensing relative to the anatomic body member so as to establish a mapincluding boundaries of the field relative to other juxtapositionedanatomic structures. A display, observable by the surgeon, is coupled tothe controller for visually displaying the virtual field boundariesbased upon the non-visible field.

[0040] In accordance with still another aspect of the invention, thereis provided a method of tactile feedback to a surgeon. This methodincludes disposing a medical implement at an anatomic body site in aposition to effect, by a manipulator, a predetermined medical procedure.Next is a step of sensing a non-visible field associated with ananatomic body member which distinguishes the anatomic body member fromthe surrounding area at the anatomic body site. Lastly, is a step ofgenerating a tactile feedback signal to the manipulator based upon aparameter of the non-visible field. This method may also include a stepof stimulating the anatomic body member; this stimulation may includeemitting a predefined electrical stimulation signal coupled to energizethe anatomic body member.

[0041] In accordance with a further aspect of the invention, there isprovided a method of identifying an internal anatomic body member thatis not visible to a surgeon by direct optical means so as to enableimproved control by the surgeon of a surgical procedure. This methodincludes sensing a non-visible field that is established and associatedwith the anatomic body member and controlling the sensing so as to beresponsive to signals representative of different locations of sensingrelative to the anatomic body member. This sensing establishes a mapincluding virtual boundaries of the sensed field relative to otherjuxtapositioned anatomic structures. Last is the step of displaying, ona display, observable by the surgeon, the anatomic body site as well asthe virtual field boundaries. This method may also include stimulatingthe anatomic body member.

[0042] In accordance with still another aspect of the invention, thereis provided a medical system that comprises a sensing device forobtaining a visual image of an anatomic body site, a medical implementand a manipulator controlled by an operator for control of the implementat the anatomic body site. A controller intercouples the sensing deviceand the manipulator and includes a mapping component for translatingpredetermined characteristics of the visual image into a signal forcontrolling action at the manipulator.

[0043] In accordance with further aspects of this medical system, thesignal for controlling action at the manipulator may comprise a forcesignal or a vibration signal. The sensing device may comprise anendoscope. The controller may control actions at the manipulator as afunction of the position of the surgical implement within the anatomicbody site. The surgical implement may comprise a surgical instrumentwith an end effector or a catheter or any other type of surgicalimplement. The manipulator may include a hand-held instrument or maycomprise a haptic interface of a tele-robotic system.

[0044] In accordance with another aspect of the invention, there isprovided a medical system that comprises a sensing device for obtaininga visual image of an anatomic body site, a surgical implement, and amanipulator controlled by a surgeon for control of the surgicalimplement at the anatomic body site. An audible device is employed inthe system along with a controller intercoupling the sensing device andthe audible device. This controller includes a mapping component fortranslating predetermined characteristics of the visual image into asignal for controlling the audible device.

[0045] In accordance with further aspects of the invention, thepredetermined characteristics may represent an outline of an anatomicbody member. The controller may control the audible device as a functionof the position of the surgical implement within the anatomic body siteand relative to the anatomic body member. The audible device is soundedwhen the surgical implement comes within a predefined proximity to theanatomic body member, so as to signal that the surgical implement is tooclose to the anatomic body member.

[0046] In accordance with still another aspect of the invention, thereis provided a medical system that comprises a surgical implement, amanipulator, controlled by an operator for control of the surgicalimplement at an anatomic body site, and a force sensor for detecting bya direct contact a force imposed on the surgical implement at theanatomic body site. Also included is a visual display observable by theoperator as well as a controller intercoupling the force sensor and thevisual display and including a mapping block for translatingpredetermined characteristics from the force sensor into a signal forcontrolling the content of display on the visual display.

[0047] In accordance with further aspects of the invention, thecontroller controls the display content as a function of the position ofthe surgical implement within the anatomic body site. The force sensormay be carried directly on the surgical implement or may be at someother defined location relative to the surgical implement. Thecontroller is responsive to a force signal from the force sensor forhighlighting an area of the visual display corresponding to a locationof the surgical implement. The highlighting may also include providingan indicia on the visual display representative of an area where a forceof greater than a predetermined threshold force is sensed.

[0048] In accordance with a further aspects of the invention, there isprovided a medical system that comprises a surgical implement, amanipulator controlled by an operator for control of the surgicalimplement at an anatomic body site, and a force sensor for detecting bydirect contact a force imposed on the surgical implement at the anatomicbody site. Also included in the system is a controller intercoupling theforce sensor and the manipulator and including a mapping block fortranslating predetermined characteristics from the force sensor into asignal for controlling vibration at the manipulator.

[0049] In accordance with further aspects of this system, the controllercontrols the manipulator as a function of the position of the surgicalimplement within the anatomic body site. The force sensor may be carriedby the surgical implement or may be at another location. The controlleris responsive to a force signal from the force sensor for controllingthe manipulator as a function of the measured force level.

[0050] In accordance with a further aspect of the invention, there isprovided a medical system that comprises a surgical implement, amanipulator controlled by an operator for control of the surgicalimplement at an body site and a force sensor for detecting by directcontact a force imposed on the surgical implement at the body site. Alsoincluded in the system is an audible device and a controller forintercoupling the force sensor and the audible device and including amapping component for translating predetermined characteristics from theforce sensor into a signal for controlling the audible device.

[0051] In accordance with further aspects of this system, the controlleris responsive to a force signal from the force sensor for controllingthe magnitude of an audible signal from an audible device as a functionof the magnitude of the force signal. The audible signal may begenerated only when the force signal exceeds a preselected threshold.

[0052] In accordance with another aspect of the invention, there isprovided a medical system comprising a surgical implement, a manipulatorcontrolled by an operator for control of the surgical implement at ananatomic body site and a sensing device for sensing a non-visible fieldgenerated from an anatomic body member disposed at the anatomic bodysite. The system also includes a visual display and a controllerintercoupling the sensing device and the visual display and including amapping component for translating predetermined characteristics of thenon-visible field from the sensing device into a signal for controllingthe content on the visual display.

[0053] In accordance with other aspects of the above system, themanipulator intercouples with the controller and the controller isresponsive to actions at the manipulator to control the surgicalimplement in carrying out a surgical procedure. The controller mayestablish on the display both a visual image of the area at the anatomicbody site as well as a virtual image representative of a boundarydefined by the mapping component and relating to the non-visible field.The non-visible field may be an electric field (which as used hereinincludes an electromagnetic field), a thermal field, a radioactivefield, an IR field, or any other type of non-visible field.

[0054] In accordance with the various systems of the invention, thesensing device may comprise a voltmeter, a thermistor, a gamma counter,an IR detector or any other type of field detecting device. Also, afurther sensing device may be used for obtaining a visual image of theanatomic body site wherein the controller translates predeterminedcharacteristics of the visual image into a signal for controllingactions at the manipulator. The signal for controlling action at themanipulator may comprise a vibration signal. Also, an audible device maybe employed wherein the controller translates predeterminedcharacteristics of the visual image into a signal for controlling theaudible device. The aforementioned system may also include a forcesensor for detecting by direct contact a force imposed on the surgicalimplement at the anatomic body site. The controller may translatepredetermined characteristics of the force signal into a signal forcontrolling the content on the visual display. The controller maytranslate predetermined characteristics of the force sensor into asignal for controlling vibration at the manipulator. If an audibledevice is employed, the controller may translate predeterminedcharacteristics from the force sensor into a signal for controlling theaudible device.

[0055] In accordance with another aspect of the invention, there isprovided a medical system that comprises a medical implement, amanipulator controlled by an operator for control of the medicalimplement at an anatomical body site, and a sensing device for sensing anon-contact field generated from an anatomic body member disposed withinthe anatomic body site. A controller intercouples the medical implement,manipulator and sensing device and includes a mapping component fortranslating predetermined characteristics of the non-contact field fromthe sensing device to a signal for controlling actions at themanipulator.

[0056] In accordance with further aspects of this system, the signal forcontrolling action may control a tactile feedback to the operator. Thesignal for controlling actions may control a vibration level at themanipulator and there may also be provided a second sensing device forobtaining a visual image at the anatomic body site. This may include anendoscope.

[0057] In accordance with still a further aspect of the invention, thereis provided a medical system that comprises a surgical implement forcarrying out a predetermined surgical procedure at an anatomic bodysite, a sensing device for sensing a non-visible field generated from ananatomic body member disposed within the anatomic body site, and anaudible device. A controller intercouples the sensing device and theaudible device and includes a mapping component for translatingpredetermined characteristics from the sensing device into a signal forcontrolling the audible device. A manipulator may be provided forcontrolling the surgical implement at the anatomic body site. Thecontroller is responsive to the magnitude of the sensed field forcontrolling the audible signal from the audible device as a function ofthe magnitude of the sensed field.

[0058] Numerous other features of the present invention should nowbecome apparent upon a reading of the following detailed description:

BRIEF DESCRIPTION OF THE DRAWINGS

[0059]FIG. 1 is a schematic block diagram illustrating an embodiment ofthe present invention;

[0060]FIG. 2 is an embodiment of a visual display according to thepresent invention;

[0061]FIG. 3 is a schematic block diagram of an alternate system inaccordance with the present invention, as applied to a hand-heldinstrument;

[0062]FIG. 4 is a schematic diagram illustrating an alternate embodimentof a device for locating a sensor;

[0063]FIG. 5 is a block diagram illustrating various relationshipsbetween sensed parameters, transformed data and signals, and appliedoutputs, according to various embodiments of the present invention; and

[0064]FIG. 6 is a schematic block diagram illustrating an alternativeembodiment of the present invention employing a nerve stimulator.

DETAILED DESCRIPTION

[0065] Visual displays have been used extensively in connection withsurgical procedures. However, these displays do not depict invisible ornon-visible fields associated with a particular anatomic body structureof interest. One embodiment of the present invention identifies thesenon-visible fields so as to provide feedback to a surgeon interface.Electrical, thermal, radioactive, IR and other non-visible fields thatsurround tissues or that are associated with other anatomic structures,are sensed with an appropriate sensor. The sensor may be disposed at adistal end of an instrument. The sensor may be employed in a roboticsystem or in association with a hand-held instrument, such as alaparascopic instrument.

[0066] The term “invisible field” or “non-visible field” refers to ameansureable parameter, but one that is not discernable to the naked eyeby a surgeon, physician, or operator, such as through the traditionaluse of an endoscopic camera at an anatomic site. Although importantembodiments of the invention relate to sensed “non-visible” fields, itis understood that other aspects of the invention extend to sensing ofvisible and/or force sensed data.

[0067] A signal from a sensing device may be amplified and preferably istransduced into a digital signal. With a controller, the digital signalcan be mapped to, for example, a tactile feedback (palpable hardness orvibration sense) provided at a surgeon's tactile (haptic) interface.

[0068] In accordance with another aspect of the invention, successivedigital signals may be stored to establish a database corresponding toan anatomic body area. The stored data may be used for establishing avirtual image or boundary on a display device for identifying, on thedisplay device, the anatomic member that has generated the sensed field.In essence, the virtual image or boundary may be overlayed on a visibleimage to identify to the surgeon an invisible anatomic body member, suchas a hidden nerve or nerve bundle.

[0069] In association with electrical (which as used herein is meant toinclude electromagnetic) fields, these electrical potentials, dipoles,and/or ionic currents that surround nerves can be measured usingmillivoltmeters. These meters may be embedded in a distal tip of aninstrument, such as the tip of a robotic finger. In this way, differingelectrical forces and dipoles can be mapped and then provided to thesurgeon as different degrees of hardness/softness, or vibration. Thesurgeon is thus able to “feel” subtle electrical potentials, dipoles andionic currents. Because these electrical fields are generally associatedwith nerves or nerve bundles, the surgeon is essentially able todetermine the position of these nerves or nerve bundles even though theyare not visible to the surgeon through a conventional optical system,such as an endoscope.

[0070] Software algorithms may be used to identify certain electricalfields. In this way the electrical “signature” of a particularneurovascular bundle, such as the facial nerve or sensory nerves, can bediscriminated from other neuro-electrical impulses. If no signature ispresent, e.g., due to denervation, then an ultra-low energy nervestimulator may be employed. FIG. 6, which will be described in furtherdetail hereinafter, illustrates a nerve stimulator for providing orenhancing a “signature”.

[0071] A stimulator is capable of emitting a predefined specificrepeating electrical signal created by a sensing computer. In this waythe stimulator energy: 1) is kept below the threshold of injury to thenerve trunk and electrical trauma to the nerve is eliminated(conventional nerve stimulators may cause injury); and 2) an electrical“signature” is created to specifically identify the particular nerve.The sensing computer can thus identify specific nerve roots and trunksby the particular electrical pattern “signature” generated by the nervestimulator.

[0072] With a unique electrical signature, a robot (for example) canthen track the nerve or neurovascular bundle and establish an algorithm(mapping) to create a virtual wall around the structure, which is thenused to prevent inadvertent transection or trauma to the structure. Theelectromagnetic signature of different neurovascular structures can beprogrammed into the robot used for a surgical dissection. As a result, avirtual conduit is placed around the neurovascular bundle such that therobot automatically dissects along this neurovascular bundle, trackingalong the bundle to the target area (e.g., tumor, aneurysm, or otherdisease state) but does not transect the neurovascular bundle or deviatefrom the path of the bundle to other anatomical structures. Thisfacilitates rapid surgical dissection in a relatively bloodless mannerand ensures that no injury occurs to the nerves along the pathway.

[0073] In a similar manner, thermal sensors may be positioned at the tipof the surgical instument to measure and relay subtle differences intemperature. These differences can discriminate, for example, an arteryfrom a vein in neurovascular bundles on the extremities. A thermistoreither amplifies the thermal signal, i.e., translate a 0.1° temperatureas a 10° temperature, or translates the thermal signal into a palpableharness/softness or vibration signal using a piezo-electric crystal inthe surgeon's haptic interface. Other critical structures such as theureters can be flushed with warm saline solution to enhance temperaturerecognition. A software algorithm coupled to a thermistor may, forexample, block a dissection in areas above a certain temperature. Inthis example, the ureters would feel “hard” relative to surroundingtissues, enabling the surgeon to avoid the ureters.

[0074] In a similar manner, a gamma-counter may be mounted to the tip ofan instrument. Following injection of a radiolabled substance,dissection and sensing of the radioactivity proceed simultaneously. Thedegree of radioactivity may be mapped to a vibrational signal, tactilyindicating to a surgeon proximity to a critical tissue site, (e.g.,lymph node, abscess pocket or other radio-tagged tissue).

[0075] In the foregoing examples, reference has been made to the sensingof certain non-visible parameters for the purpose of providing real-timeforce or vibration feedback to the haptic interface. These sameparameters may also be sensed, in either a sequential or scanning mode,and the values stored in order to establish a database, usedsubsequently for mapping a virtual image onto a display device.

[0076] Reference is now made to the drawings, and in particular FIGS. 1and 2. FIG. 1 illustrates a block schematic diagram of one embodiment ofthe present invention. FIG. 2 shows details of a visual displayillustrating, by virtual field boundaries, the location of invisiblestructures.

[0077]FIG. 1 illustrates as a representative internal body site ofinterest a visible tissue 1 having extending therethrough a nerve 2.Although the nerve is shown in FIG. 1 it is generally not visible on thedisplay 13. FIG. 1 shows the nerve 2 inside the tissue 1, in dottedoutline to indicate the non-visible nature of the nerve.

[0078]FIG. 1 also illustrates the distal end of a surgical instrument 3having a tool or end effector, such as the gripping jaws 4 illustratedin FIG. 1. An endoscope 14 is also shown proximate to the tissue 1. Theendoscope 14 is shown extending through an incision 18 illustrated indotted line in FIG. 1. The endoscope typically carries a camera fordetecting a visual image at the tissue 1, which is shown on the display13.

[0079] In FIG. 1, associated with the nerve 2, is an electric field Fshown in dashed lines. This field is considered as a non-visible field.In this particular example it is an electric field but also, the sensingin accordance with the present invention can be thermal, radioactive, IRand other types of fields associated with anatomic structures, andincluding self-excited fields.

[0080]FIG. 1 shows a sensor 5 on the tool at the tip of the instrument3. Here the sensor 5, in sensing the location of a nerve 2, may be avoltmeter or, more appropriately, a micro or millimeter range voltagemeter. The output from the sensor 5 couples to a sensing system 8 which,in turn, couples to a computation system 9. A drive system 7 is coupledfrom the computation system 9 to the instrument 3 for the control of theoperation of instrument 3.

[0081] Also depicted in FIG. 1 is display system 13 interconnecting withthe computation system 9. A second drive system 10 couples to thecomputation system 9 and interfaces with a haptic interface 11. Adjacentthe haptic interface, here an operating handle 12, is illustrated asurgeon's hand.

[0082] Thus, the surgeon can carry out manipulations at the hapticinterface 11 in order to control, by way of drive system 10, computationsystem 9, and drive system 7, the instrument 3 conducting some medicalprocedure. The control may include multiple degrees-of-freedom ofcontrol of the instrument, one of which is opening and closing of thejaws or grippers 4. While the surgeon is carrying out suchmanipulations, there is a visual image of the tissue 1 on the display13. However, this visual image does not include an image of the nerve 2because this is essentially invisible on the display 13. However, inaccordance with the invention by detecting the non-visible electricalfield F, via the sensor 5, the computation system 9 can map the detectedfield to the display 13, as shown in FIG. 2.

[0083]FIG. 2 illustrates, on the display 13, a visual image of thetissue 1. The display also depicts a virtual image or boundary,designated by polygons 21, that correspond in position generally to theplacement of nerve 2 within the associated tissue 1. In this way, if theend effector 4 is performing certain tasks associated with the tissue 1,including dissection of the tissue, the virtual image area representedby the polygons 21 may be avoided.

[0084] The virtual image area displayed in FIG. 2 may be developed by anoperator manually moving the instrument 3 (and sensor 5) about an areaproximate to the tissue 1. As the instrument/sensor is moved, differentvoltages are sensed relative to the field F as the end effector is movedfrom position to position. Alternatively, after the instrument is placedthrough the incision, movement of the instrument/sensor could becontrolled by a computer to transition about the tissue 1 in asufficient number of locations so as to create a “map”, and from thatmap, a database. If the instrument is under computer control, thiscontrol can be established in a predetermined pattern or it can be aform of random control. The computation system 9 may store (in adatabase) a field strength signal associated with each of the locationsvisited by the sensor as it is either moved by the surgeon orautomatically under computer control. An algorithm of the computersystem 9 then looks at the data and determines the boundaries,establishing the polygons 21 illustrated in FIG. 2. These representareas to be avoided, in this particular example, as they overlie an areawhere a nerve appears and where dissection is not to occur.

[0085] The computation system 9 receives several pieces of data used indetermining the content of the display 13. First, from the endoscope 14,a visual image data set is generated by digitizing the received opticalpattern into well-known “pixels” representative of the whole area beinglooked at. These pixels can be stored on an updating basis by locationand intensity, with intensity levels corresponding to each and everylocation being displayed to provide the actual (visual) image at thesite of interest. Next, system 9 receives a signal from for example alocation sensor identifying the location of the instrument end effector(and thus field sensor 5) at all times. The location of the end effectoror tool is well-known in telerobotic systems, particularly as it relatesto manipulations at the master haptic interface.

[0086] In addition, from the sensor 5 there is provided to computationsystem 9, for a plurality of sites, location and intensity of thedetected non-visible field. In this example of using a voltmeter todetect electrical activity from a nerve or nerve bundle, by scanning thearea of interest (either manually or under computer control) one canestablish intensity levels for all sensed locations. These levels can beessentially overlayed on the visual image display as polygons 21illustrated in FIG. 2. For example, the apex of each polygon mayrepresent points of the same intensity. The polygons may be in the formof triangles of a size as depicted in FIG. 2, or may be smallerdepending upon the resolution of the detection system.

[0087] The database contains stored values of the sensed parameter, asto location and intensity. If scanning was performed under computercontrol, there is a field strength stored for each location (pixel)visited during the scan. A control algorithm in computation system 9then simply examines the database and determines boundaries, such asrepresented by the polygons 21 in FIG. 2. In a simple form, thealgorithm may utilize threshold levels for demarcation. For example,levels over a certain predetermined value are highlighted. Thishighlighting could be by color variation or other indicia on thedisplay, such as by establishing a “fog” or “blur” in those areas abovethe detected threshold.

[0088] There may be provided certain synchronization signals at thecomputation system 9 to control both the visual image and the virtualboundaries; such synchronization is within the knowledge of the personin the art. The display device 13 is typically controlled bysynchronization signals and here there is simply a case of coordinatingvisual image signals with virtual image signals.

[0089] In addition to establishing a virtual image, the system may alsoprovide feedback to the haptic interface 11. For example, as the sensor5 gets closer to the nerve 2, the field strength increases. Thismeasured strength, sensed by the sensor, is coupled by way of thesensing system 8 to the computation system 9, and from there to drivesystem 10 and haptic interface 11. This feedback can be used to indicateat the handle 12 that the distal end of the instrument 3 is approachingan area where there is a nerve, so that dissection should be avoided inthat area. This control is on a real-time basis.

[0090] Thus, as the surgeon manipulates the handle 12, if the instrumenttip gets close to a nerve 2, there will be a feedback by way of thesensing system 8 so as to provide one or more of a vibration to thehandle 12 and a force feedback to the handle 12 in order to inhibit orlessen further action at the end effector. If the end effector is ascalpel or scissors, this would impede or lessen further action andsignal to the surgeon that he is in the vicinity of a nerve and must becareful to not cause a dissection of the nerve. This control may beexercised using threshold levels. Alternatively, the feedback may beprovided as an audible alarm.

[0091] In connection with the system described in FIGS. 1 and 2, it isnoted that the system operates so as to essentially “warn” a surgeon(provide an alarm) of proximity to a nerve or other critical anatomicstructure (an avoidance area). This aspect of the invention also appliesto establishing a “landmark”, for example, so as to highlight or even toguide the instrument to the landmark area. As a further example, mentionhas been made previously to use of a gamma counter for sensing andlocating a critical tissue site such as a lymph node. The sensedradioactivity can be used to pinpoint the location of this anatomicstructure for the purpose of a surgical procedure.

[0092] Reference is now made to the block schematic diagram of FIG. 3,disclosing an alternative embodiment utilizing a hand-held instrument.The instrument 30, which may be a laparascopic instrument, has at oneend an operating handle 32 (as the haptic interface) and at the otherend an end effector 34. The end effector 34 is illustrated in proximityto a tissue 40. Associated with the end effector 34 is a sensor 5 thatmay be, for example, a voltmeter, thermistor or radioactive detector. Asignal line 36 couples from the sensor to the sensing system 8.

[0093]FIG. 3 depicts an anatomic wall 42 having an incision 43 forreceiving a shaft 44 of instrument 30. The instrument shaft may beguided through the incision by a trocar 46, as illustrated. Under normaloperating conditions, the surgeon can manipulate the handle 32 toactuate the end effector 34 and can, under manual control, move theinstrument in multiple degrees-of-freedom in carrying out a surgicalprocedure.

[0094] In addition to the sensing system 8, FIG. 3 also includescomputation system 9 and drive system 7. In this embodiment the drivesystem 7 generates a control signal on line 48 for feeding back to theinstrument 32 a vibration signal as an indication to the surgeon at thehandle 32 that the end effector is coming into close proximity to asensed non-visible field F. When the end effector 34 comes insufficiently close proximity to the site having the associated field F,the sensing system 8 senses this condition and the computation system 9along with drive system 7 causes a signal on line 48 to create avibration at the handle 32. The magnitude of the vibration may be afunction of the proximity of the end effector to the field. As in theprevious example given, the field may be generated from a nerve or nervebundle. FIG. 3 also illustrates an alternate form of feedback to thesurgeon, where the computation system 9 has an output coupling to anaudio speaker 50 for providing an audible signal to the surgeon of aproximity of the end effector to the aforementioned field.

[0095] In FIGS. 1-3, the sensor is disposed at the very tip of theinstrument, more specifically at the end effector. However, inaccordance with other embodiments the sensor may be disposed at otherpositions on the instrument, or could even be at a position other thanon the instrument. In this regard, FIG. 4 shows a portion of aninstrument 60 having an arm 62, a wrist section 64 and end effector 66.The instrument 60 is illustrated in proximity to a tissue 68 that may beconsidered as having a nerve or nerve bundle close to its surfaceemanating a non-visible field F 70. The arm 62 is carrying a sensor 72at a predetermined position along the arm. Here, the purpose may be toavoid having the arm 62 contact the tissue 68. By disposing the sensor72 on the arm 62, a feedback signal can be provided to the surgeon as tothe proximity between the instrument arm 62 and the tissue 68.

[0096] The mapping aspect of the present embodiments relate to thetranslation of certain sensed parameters into actions or displays forimproved surgeon control of certain surgical procedures. The control canbe provided by giving the surgeon an improved display to observe as heoperates, a visual image along with a virtual image as described herein.The control may also include feedback to the haptic interface to providethe surgeon with tactile or audible feedback based upon sensednon-visible parameter.

[0097] Various mapping or translation techniques are illustrated in theblock diagram of FIG. 5. FIG. 5 shows a sensing side at 100 and adisplay or control side at 102. Various system blocks extend betweenthese two sides to provide the mapping. Specific examples followhereinafter. It is understood that systems can be readily designed thatincorporate only some of the blocks shown in FIG. 5.

[0098] On the sensor side 100 of FIG. 5 there are depicted a number ofdifferent sensing devices, each coupling to a measurement and transformsystem 110. Often, only one sensor is employed in a specific embodiment.These sensors include an endoscope 112, a force sensor 114, a voltmeter116, a thermal sensor 118, an IR detector 120, and a radioactivedetector 122. The endoscope 112 can provide a visual image (at display142) of the anatomic body site of interest. The force sensor 114 may beany well-known type of force sensor that provides an indication of acontact force when disposed for example at the end effector of theinstrument. The other sensors 116, 118, 120 and 122 are non-contact andnon-visible field sensors. The voltmeter 116 may comprise amilli-voltmeter or micro-voltmeter. The thermal sensor 118 may include athermistor. The radioactive detector 122 may comprise a gamma counter.

[0099] In FIG. 5, the measurement system 110 may provide some type ofsignal transformation or processing. Multiplexing may be provided tohandle the various sensed signals. Two basic data paths are shownconnecting from the output of the measurement system 110—one path atline 130 to controller 136 that represents current (“real time”) senseddata from one of the sensors, and another path at line 132 to database138 that represents data, over time, to be stored in the historicdatabase 138. The output of database 138 also couples via line 134 tocontroller 136. The controller 136 couples to and controls devices suchas the illustrated visual display 142, audible device 144, and hapticinterface force feedback device 140.

[0100] In FIG. 5 the side 100 may be considered as the slave side orstation, while the side 102 may be considered as the master side orstation. Thus, in FIG. 5 there is shown at side 102 a haptic interfacewhich has the ability of receiving tactile (force or vibration) feedbacksignals indicative of the sensing of certain fields at the slave side.Also illustrated in FIG. 5 is the visual display 142 and the audibledevice 144.

[0101] In FIG. 5, one or both of the controllers 110 and 136 provide thesignal mapping and contain one or more pre-stored algorithms todetermine virtual boundaries (polygons 21) from the historic data in thedatabase 138. The controller(s) also control feedback to the hapticinterface 140, such as from the sensors, and may or may not take intoaccount historic data. Of course, the controller(s) also control actioninitiated by the operator at the haptic interface to the instrument.

[0102] In FIG. 5 there may also be applied certain modeling techniques.These may be considered as applied at one or more of the controllers 110and 136. Modeling techniques can be used to transform data at the masterside based upon certain established models. For example, modeling may beused to account for tissue movements based upon instrument actions.Modeling may be particularly advantageous for anatomic sites where it isanticipated that tissues or organs may move upon interaction with thesurgical instrument. This involves spatial modeling. There may also becircumstances, such as are involved in cardiac procedures, when temporalmodeling can be applied. Models may be used that account for bothspecial and temporal parameters. A force feedback from a force sensor(see FIG. 5) may be used as a control element in controlling the finiteelement modeling. In temporal modeling the control algorithm may bebased primarily on an EKG waveform.

[0103] In other embodiments of the invention, rather than using only onetype of sensor for performing a particular procedure, multiple sensorsmay be employed. These multiple sensors could be supported on aninstrument carrying one or more end effectors, or could be supportedtogether on a separate instrument or probe. Alternatively, each sensorcould be supported by its own probe or instrument. For example, inperforming a sentinel node biopsy, a first sensor may be used toestablish a virtual image of pertinent artery, vein and nervestructures, and a second sensor may be used to detect radio activity atthe pertinent mode, thus generating a second virtual image or boundaryof the node location. These virtual images may be overlayed in a visualdisplay to assist the surgeon in quickly and reliably carrying out thesurgical procedures.

[0104] The sensing that is employed, such as in association with nervesor nerve bundles, typically relies upon an inherent electric fielddisposed about the nerve or nerve bundle. However, there may becircumstances in which such a field is not of a sufficient strength.Thus, in the embodiment of FIG. 6, there is added to the system a nervestimulator 150. FIG. 6 schematically illustrates a nerve 152, a gripper154 or other end effector construction, and a signal transducer 156coupled with the gripper and receiving signals from a sensor 158disposed on one of the grippers. The signals that are outputted from thesignal transducer 156 couple to a control computer 160 that alsointerfaces with the haptic interface 162.

[0105] The system of FIG. 6 employs an ultra-low energy nerve stimulator150. The stimulator is capable of emitting a predefined specific andpreferably repeating electrical signal that is controlled from thecomputer 160. The output of the stimulator 150 is kept below thethreshold of injury to the nerve trunk 152. This eliminates anyelectrical trauma to the nerve. Furthermore, this system creates anelectrical “signature” to specifically identify the particular nerveconstruction. The sensing computer 160 can thus identify specific nerveroots and trunks by the particular electrical pattern “signature”generated by the nerve stimulator 150.

[0106] The following are further examples of various aspects of theinvention. Several examples of different mapping concepts are set forth,supported by descriptions found in FIGS. 1-6. These mapping concepts maybe practiced in either a robotic or non-robotic system.

[0107] Mapping Visible Data to Haptic Interface

[0108] This may be embodied in a medical system comprising a sensingdevice for obtaining a visual image of an anatomic body site, a surgicalimplement, and a manipulator controllable by a surgeon for control ofthe surgical implement at the anatomic body site. The sensing device maycomprise the endoscope 112 illustrated in FIG. 5. The manipulator may bethe one illustrated in FIG. 1. The system includes a controller, such asthe controller 136 in FIG. 5. The controller intercouples the sensingdevice and the manipulator and includes a mapping component fortranslating predetermined characteristics of the visual image into asignal for controlling action at the manipulator. For example, there maybe certain aspects of the visual image in which the visual data issynthesized to identify certain locations of the visual display that areto be highlighted and thus avoided by the surgeon. When the instrumentis at these locations then the controller provides a force feedbacksignal to the manipulator, or haptic interface 140 illustrated in FIG.5. The feedback signal may be in the form of a tactile or audiblesignal. Preferably, the feedback signal is a signal that the surgeon“feels” (tactile) at the haptic interface essentially signaling to thesurgeon that the instrument is at an area that is to be avoided. Thisfeedback to the haptic interface may supplement the surgeon's actualdirect viewing of a display, such as the display 142 illustrated in FIG.5 or the display 13 illustrated in FIG. 1.

[0109] Accordingly, this type of mapping identifies certaincharacteristics of the visual display, typically through an endoscope,and converts the sensed data to a feedback control signal at the hapticinterface. The parameters that are identified on the display may be, forexample, arteries, veins or other anatomic structures. Even though thesestructures may be visible to the surgeon through the display, there isan added advantage of providing this feedback to the haptic interface tosupplement what the surgeon views at the display.

[0110] This form of mapping identifies visual boundaries such as bydifferences in contrast or gray scale and converts these boundaries totransition locations where signals are fed back to the master interfaceto signal (to the surgeon) areas that are to be avoided or targeted.

[0111] Mapping Visible Data to Vibration

[0112] Such a medical system may employ many of the same components asdiscussed with regard to the mapping of visual data to a hapticinterface. However, instead of a feedback signal at the hapticinterface, the control signal may be provided to the manipulator andcomprise a vibration signal. In this regard, FIG. 3 shows a vibrationsignal on line 48 coupling from drive system 7 to manipulator (handle)32. The only difference here is that in FIG. 3 one is sensing anon-visible field, while in this example one is providing a vibrationsignal based upon sensing a visual field.

[0113] Mapping Visible Data to Audible Device

[0114] This may be embodied in a medical system that comprises a sensingdevice for obtaining a visual image of an anatomic body site, a surgicalimplement and a manipulator controlled by a surgeon for control of thesurgical implement at the anatomic body site. The system also includesan audible device. The sensing device may be the endoscope 112 in FIG.5. The manipulator may be the haptic interface 11 in FIG. 1. The audibledevice may be device 144 in FIG. 5. The medical system also includes acontroller, such as controller 136 in FIG. 5. This controllerintercouples the sensing device and the audible device and includes amapping component for translating predetermined characteristics of thevisual image into a signal for controlling the audible device.

[0115] Accordingly, in this form of mapping, certain characteristics ofthe visual image are interpreted by the controller and if the instrumentgoes within identified locations, an audible alarm is activated. Thisindicates to the surgeon that the instrument is approaching or within azone of the anatomic body site where either the surgeon has to be morecareful or that should be avoided. Alternatively, the alarm may identifyan area to be targeted.

[0116] The predetermined characteristics of the visual signal mayrepresent an outline of an anatomic body member. The controller controlsthe audible device as a function of the position of the surgicalimplement within the anatomic body site and relative to the anatomicbody member. The audible device is sounded when a surgical implementcomes within a predefined proximity to the anatomic body member, so asto signal that, for example, the surgical implement is too closethereto.

[0117] Mapping Force Sensor to Visual Display

[0118] A medical system that embodies this mapping concept may comprisea surgical implement, a manipulator controlled by an operator forcontrol of the surgical implement at an anatomic body site, and a forcesensor for detecting a force imposed on the surgical implement at theanatomic body site. Refer to FIG. 5 for a force sensor 114. Such amedical system may also include a visual display observable by theoperator and a controller, such as controller 136 of FIG. 5 thatintercouples the force sensor and visual display. The controllerincludes a mapping component or block for translating predeterminedcharacteristics from the force sensor into a signal for controlling thecontent of information of the visual display. The controller controlsthe display content as a function of the position of the surgicalimplement within the anatomic body site. Preferably the force sensor iscarried at the tip end of the surgical implement. The controller isresponsive to a force signal from the force sensor for highlighting anarea of the visual display corresponding to a location of the surgicalimplement.

[0119] In accordance with this form of mapping, an area on the displaycan be “highlighted” to indicate an area where the detected force hasexceeded a predetermined threshold. Such mapping may be employed on a“real time” basis or the surgical implement may be moved by the surgeonthroughout an anatomic body site, or under computer control to generatea map of forces that are detected. This map may then be illustrated onthe display in a variety of different ways such as by the use of acontrasting color, a “flashing” on the display, or the use of otherindicia on the display such as illustrated in FIG. 2.

[0120] Mapping Force Sensor to Vibration

[0121] A medical system embodying this form of mapping may comprise asurgical implement, a manipulator controlled by an operator for controlof the surgical implement at an anatomic body site, and a force sensorfor detecting a force imposed on the surgical implement at the anatomicbody site. A controller intercouples the force sensor and themanipulator and includes a mapping block for translating predeterminedcharacteristics from the force sensor into a signal for controllingvibration at the manipulator. Again, reference may be made to FIG. 3showing a feedback vibration signal. In accordance with this form ofmapping the vibration signal may be activated in accordance with theoutput of the force sensor.

[0122] The controller controls the manipulator as a function of theposition of the surgical implement within the anatomic body site. Theforce sensor may be carried at the tip of the surgical implement. Thecontroller is responsive to a force signal from the force sensor forcontrolling the manipulator as a function of the measured force level.

[0123] A force sensor map may also be developed by the surgeon movingthe surgical implement with its mounted sensor about an anatomic bodysite, or by creating movement by way of computer control. This canestablish a gradient over an anatomic body site of force levels that canbe used to control, via the controller, the vibration signal coupled tothe manipulator.

[0124] Mapping Force Sensor to Audible Device

[0125] A medical system embodying this type of mapping may comprise asurgical implement, a manipulator controlled by an operator for controlof the surgical implement at an anatomic body site, and a force sensordetecting a force imposed on the surgical implement at the anatomic bodysite. This system also includes an audible device such as the device 144shown in FIG. 5 and a controller, such as a controller 136 shown in FIG.5. The controller intercouples the force sensor and the audible deviceand includes a mapping block for translating predeterminedcharacteristics from the force sensor into a signal for controlling theaudible device. The controller is responsive to a force signal from theforce sensor for controlling the magnitude of the audible signal fromthe audible device as a function of the magnitude of the force signal.The audible signal may be generated only when the force signal exceeds apredetermined threshold or the audible signal may have an outputmagnitude that directly or inversely relates to the force signalmagnitude.

[0126] In accordance with this form of mapping, a detected force signalis coupled to the controller and from there to an audible device toindicate an audible signal to a surgeon when a predetermined force issensed. This preferably occurs in “real time”.

[0127] Mapping Non-Visible Field to Visual Display

[0128] A medical system that embodies this mapping concept may comprisea surgical implement, a manipulator controlled by an operator forcontrol of the surgical implement at an anatomic body site and a sensingdevice for sensing a non-visible field generated from an anatomic bodystructure disposed within the anatomic body site. The system may alsoinclude a visual display and a controller. With regard to the sensingdevice, reference is made to FIG. 5 illustrating a number of differentforms of sensing devices for measuring the non-visible field. Thecontroller intercouples the sensing device and the visual display andincludes a mapping block (see FIG. 5 and controller 136) for translatingpredetermined characteristics of the non-visible field from the sensingdevice into a signal for controlling the content on the visual display.

[0129] The manipulator, such as the handle 12 illustrated in FIG. 1 aspart of the interface 11, intercouples with the computation system 9 inFIG. 1. The controller is responsive to actions at the manipulator tocontrol the surgical implement in performing a surgical procedure. Thecontroller establishes on the display both a visual image of the area atthe anatomic body site as well as a virtual image representative of aboundary defined by the mapping block. The polygons 21 in FIG. 2illustrate a boundary on the visual display representative of an areathat is actually invisible to the surgeon observing the display (such asdisplay 13 illustrated in FIG. 1).

[0130] The non-visible field may be any one or more of a variety offields. Refer to FIG. 5 and inputs to the measuring system 110 such asthe voltmeter 116, thermal sensor 118, IR detector 120 or radioactivedetector 122. Thus, these fields may be, but are not limited to,electrical, electromagnetic, thermal, radioactive or an IR field.

[0131] This mapping enables the sensing of parts of anatomic structuresthat are not directly visible to the surgeon. This mapping enables aconversion of a non-visible field, into a visible (virtual) boundary ona display that the surgeon is observing. This boundary, such as thatillustrated in FIG. 2 by the polygons, is essentially a virtual conduitthat is placed about the anatomic structure such as a neurovascularbundle. In this way, the surgeon can control dissection so that itoccurs outside of and along these neurovascular bundles, tracing alongthe bundle to the target area without any transection of theneurovascular bundle. This facilitates rapid surgical dissection in arelatively bloodless manner and that insures no injury occurs to thenerves along the pathway.

[0132] Mapping Non-Visible Field to Haptic Interface

[0133] A medical system that embodies this mapping may comprise asurgical implement, a manipulator controlled by an operator for controlof the surgical implement at an anatomic body site, and a sensing devicefor sensing a non-visible field generated from an anatomic bodystructure disposed within the anatomic body site. Suitable sensingdevices are illustrated in FIG. 5. A controller intercouples thesurgical implement, manipulator and sensing device and includes amapping block for translating predetermined characteristics of thenon-visable field from the sensing device into a signal for controllingactions at the manipulator. The signal for controlling actions usuallycontrols a tactile feedback to the operator.

[0134] In accordance with this form of mapping non-visible fields aresensed. A feedback occurs to the master interface to feedback a tactilesignal to the surgeon indicating that the instrument is now at alocation that corresponds with a certain field intensity. This form oftactile feedback is carried out preferably in “real time”.

[0135] Mapping Non-Visible Field to Vibration

[0136] A medical system that embodies this form of mapping may comprisea surgical implement, a manipulator controlled by an operator forcontrol of the surgical implement at an anatomic body site, and asensing device for sensing a non-visible field generated from ananatomic body structure disposed within the anatomic body site. Acontroller is provided intercoupling the surgical implement, manipulatorand sensing device and includes a mapping block for translatingpredetermined characteristics of the non-visible field from the sensingdevice into a vibration signal coupled to the manipulator.

[0137] Mapping Non-Visible Field to Audible Device

[0138] A medical system embodying this form of mapping may comprise asurgical implement for carrying out a predetermined surgical procedureat an anatomic body site, a sensing device for sensing a non-visiblefield generated from an anatomic body member disposed within theanatomic body site, and an audible device. The audible device isillustrated as device 144 in FIG. 5 and the sensing device may be thesensing devices illustrated in the block diagram of FIG. 5. Acontroller, such as controller 136 in FIG. 5 intercouples the sensingdevice and the audible device and includes a mapping block fortranslating predetermined characteristics from the sensing device into asignal for controlling the audible device. The controller is responsiveto the magnitude of the sensed field for controlling the audible signalas a function of the magnitude of the sensed field.

[0139] With regard to the mapping, FIG. 2 shows polygons that demarcatean area of an underlying invisible structure. Where this anatomicstructure is a nerve or nerve bundle, detection of an electric fieldassociated with the nerve or nerve bundle can be accomplished by“scanning” so as to establish the map and the polygons illustrated inFIG. 2. This scanning can occur by virtue of the surgeon moving thesurgical instrument about the anatomic body site with the controllerdetecting successive positions, storing these positions and establishinga map from the stored values based upon the intensity of the signal.Alternatively, this mapping may occur automatically under computercontrol.

[0140] Reference has been made to medical and surgical implements andinstruments. Catheters or any other type of medical instrumentation orguide device can be used in accordance with the present invention.

[0141] Another system for computer-controlled manipulation of a medicalinstrument, which may be useful in various embodiments of the presentinvention, is described in U.S. Pat. No. 6,197,017, issued Mar. 6, 2001to Brock and Lee, entitled “Articulated Apparatus for TelemanipulatorSystem,” which is also incorporated by reference in its entirety.

[0142] Having now described a number of embodiments of the presentinvention, it should be apparent to those skilled in the art thatnumerous other embodiments and modifications thereof are contemplated asfalling within the scope of the present invention, as defined by theappended claims.

1. A system for generating a display of a body structure comprising: asensor positionable at an internal body site for sensing a non-visiblefield of a body structure at the site and generating a sensor signalindicative of the field; a transformation system for transforming thesensor signal into virtual image data; a source of visual image data forthe site; and a visual system enabling combined display of the visualimage data and the virtual image data.
 2. The system of claim 1, whereinthe non-visible field is at least one of an electrical field, a thermalfield, an infrared field and a radioactive field.
 3. The system of claim1, wherein the sensor is at least one of a voltage sensor, a thermalsensor, an infrared sensor, and a radioactive sensor.
 4. The system ofclaim 1, wherein the sensed field is associated with at least one of anerve, nerve bundle and vascular vessel.
 5. The system of claim 1,further including: a stimulator emitting a stimulation signal for atleast one of enhancing and generating the field.
 6. The system of claim1, further including: a tele-robotic system having a slave station witha medical implement and a master station with a user interface at whichuser input occurs to control the medical implement, wherein the visualsystem is provided at the master station.
 7. The system of claim 1,wherein the transformation system includes, for at least one selectsite, an associated signature value of the virtual image data.
 8. Thesystem of claim 1, wherein the virtual image data defines a boundary ofthe field.
 9. The system of claim 5, wherein the stimulator emits atleast one predefined stimulation signal for an associated body site. 10.A method for generating a display of a body structure comprising:sensing at an internal body site a non-visible field of a body structureat the site and generating a sensed signal indicative of the field;transforming the sensed signal into virtual image data; providing visualimage data for the site; displaying in combination the visual image dataand the virtual image data.
 11. A system for obtaining virtual imagedata of a body structure comprising: a computer-controlled instrumentfor positioning a sensor at an internal body site; the sensor sensing anon-visible field of a body structure at the site and generating asensor signal indicative of the field; and a transformation system fortransforming the sensor signal into virtual image data.
 12. The systemof claim 11, further including: a visual system enabling a display ofthe virtual image data.
 13. The system of claim 12, further including: atele-robotic system having a slave station with a medical implement anda master station with a user interface at which user input occurs tocontrol the medical implement, wherein the visual system is provided atthe master station.
 14. The system of claim 11, wherein the non-visiblefield is at least one of an electrical field, a thermal field, aninfrared field and a radioactive field.
 15. The system of claim 11,wherein the sensor is at least one of a voltage sensor, a thermalsensor, an infrared sensor, and a radioactive sensor.
 16. The system ofclaim 11, wherein the sensed field is associated with at least one of anerve, nerve bundle and vascular vessel.
 17. The system of claim 11,further including: a stimulator emitting a stimulation signal for atleast one of enhancing and generating the field.
 18. The system of claim17, wherein the stimulator emits at least one predefined stimulationsignal for an associated body structure.
 19. The system of claim 11,wherein the transformation system includes, for at least one bodystructure, an associated signature value of the field.
 20. The system ofclaim 11, wherein the virtual image data defines a boundary of thefield.
 21. The system of claim 11, wherein the sensor is disposed at adistal end of a medical instrument.
 22. The system of claim 21, furtherincluding: a medical implement also disposed at the distal end of theinstrument.
 23. The system of claim 22, wherein the medical implementincludes a tool used to perform a medical procedure at the site.
 24. Thesystem of claim 11, further including: a hand-engageable user interfaceproviding input to a computer for positioning of the instrument.
 25. Thesystem of claim 24, wherein the user interface is part of a tele-roboticmaster station.
 26. The system of claim 12, wherein the virtual imagedata is provided in the display as at least one of a landmark area andan avoidance area.
 27. The system of claim 11, wherein thetransformation system includes: modeling data for associating thevirtual image data with locations at the site.
 28. A method forobtaining virtual image data of a body structure comprising: positioningby computer control a sensor at an internal body site; sensing anon-visible field of a body structure at the site and generating asensed signal indicative of the field; and transforming the sensedsignal into virtual image data.
 29. A system for controllingmanipulation of a medical implement comprising: a sensor positionable atan internal body site for sensing a non-visible field of a bodystructure at the site and generating a sensor signal indicative of thefield; a transformation system for transforming the sensor signal into afeedback signal; a control system, including a haptic user interface,for manipulating a medical implement at the site, the control systemreceiving the feedback signal and in response thereto providing atactile signal at the user interface.
 30. The system of claim 29,wherein the tactile signal is at least one of force and vibration. 31.The system of claim 30, wherein the tactile signal is implemented at theuser interface as at least one of force and vibration.
 32. The system ofclaim 29, wherein the user interface is at least one of a hand-heldholder for the medical implement and a tele-robotic interface for themedical implement.
 33. The system of claim 29, wherein thetransformation system enables real-time implementation of the feedbacksignal.
 34. A method for controlling manipulation of a medical implementcomprising: sensing a non-visible field of a body structure at aninternal body site and generating a sensed signal indicative of thefield; transforming the sensed signal into a feedback signal; andutilizing the feedback signal to provide a tactile signal at a hapticuser interface for controlling manipulation of a medical implement atthe site.
 35. A system for controlling manipulation of a medicalimplement comprising: a sensor positionable at an internal body site forsensing a visual image of a body structure at the site and generating asensor signal indicative of the image; a transformation system fortransforming the sensor signal into a feedback signal; a control system,including a haptic user interface, for manipulating a medical implementat the site, the control system receiving the feedback signal and inresponse thereto providing a tactile signal at the user interface. 36.The system of claim 35, wherein the tactile signal is at least one offorce and vibration.
 37. The system of claim 36, wherein the tactilesignal is implemented at the user interface as at least one of force andvibration.
 38. The system of claim 35, wherein the user interface is atleast one of a hand-held holder for the medical implement and atele-robotic interface for the medical implement.
 39. The system ofclaim 35, wherein the transformation system enables real-timeimplementation of the feedback signal.
 40. A method for controllingmanipulation of a medical implement comprising: sensing a visual imageof a body structure at an internal body site and generating a sensedsignal indicative of the image; transforming the sensed signal into afeedback signal; and utilizing the feedback signal to provide a tactilesignal at a haptic user interface for controlling manipulation of amedical implement at the site.